Insect and other small object image recognition and instant active response with enhanced application and utility

ABSTRACT

A device for detecting offending objects such as insects on substrates such as on leaves, plants, in fluid or over a large area. The device has a microscope lens which magnifies a portion of the substrate and sends an image of the substrate portion to an image recognition system. If the image recognition system detects the presence of an offending object—further steps are taken to remove, neutralize or mark the area where the offending object is located for subsequent action. The device may be mounted on a movable platform such as drone for active surveillance and action over a large area.

RELATED APPLICATIONS

This application is a Continuation In Part of U.S. patent applicationSer. No. 14/505,430, filed Oct. 2, 2014, which is to issue as U.S. Pat.No. 9,053,528 on Jun. 9, 2015, which itself is a Continuation in Part ofU.S. patent application Ser. No. 13/542,416 filed on Jul. 5, 2012, whichissued as U.S. Pat. No. 8,855,374 on Oct. 7, 2014—the contents of whichare incorporated herein. This Application also claims the benefit ofU.S. Prov. Appl. Ser. No. 62/087,049 filed on Dec. 3, 2014—the contentsof which are incorporated by reference herein.

FIELD OF THE INVENTION

The current invention relates to the field of insect control and qualitycontrol, more specifically to a novel method of rapidly detecting anddisposing of insects or other undesirable substances.

BACKGROUND OF THE INVENTION

The decreased use of pesticides on the one hand and the decreasedeffectiveness of those which are in use has resulted in a disturbingproliferation of insects in food and in the home. Moreover, insects orother invading species frequently infest orchards or similar crops.

The invention described herein makes it possible to detect andimmediately dispose of thrips, mites and aphids, among other insects,including those invisible to the untrained and unpracticed eye.

Many such insects are very difficult to dislodge by washing and areresistant to pesticides. Large scale efforts such as by heating oftenharms the food or other material to be cleansed.

SUMMARY OF THE INVENTION

The invention disclosed herein is a computer image analysis system,which magnifies an image of a substrate to be checked for bugs or smallother small objects and is trained to recognize various bugs commonlyassociated with such substrates to be checked. If a bug is identifiedany of various action steps are taken in different embodiments of theinvention, including removing the bug by way of an action headassociated with the imaging device. In other embodiments of theinvention, the system is positioned on a movable platform to scan a widearea for offending objects and to perform mitigation actions once anoffending object is detected.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic side view of a scanning device disposed above asubstrate to be checked according to an embodiment of the invention.

FIG. 2 is a flow chart showing software processing steps according to anembodiment of the invention.

FIG. 3 show a side perspective view of a scanning device incorporatedwith a drone according to an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will now be described withreference to the above-identified figures. However, the drawings and thedescription herein of the invention are not intended to limit the scopeof the invention. It will be understood that various modifications ofthe present description of the invention are possible without departingfrom the spirit of the invention. Also, features or steps describedherein may be omitted, additional steps or features may be included,and/or features or steps described herein may be combined in a mannerdifferent from the specific combinations recited herein withoutdeparting from the spirit of the invention, all as understood by thoseof skill in the art.

In one preferred embodiment, a lens is used to point at a leaf oflettuce and capture an enlarged image thereof. The image may be storedin digital memory for later analysis or it may be analyzed in real time.In either case, the image is sent to a processor that is trained torecognize the general characteristics and color of the lettuce, andwhich is also trained to recognize physical characteristics and featuresof insects typically found on lettuce. The image is magnified so thatthe presence of the insect, even if well hidden, will be identified bycomparison with a library of insects stored in memory. In oneembodiment, the processor does not positively identify a bug or otherforeign object, but it may recognize the object as foreign. Forinstance, a processor may contain parameters of acceptable color valuesor hues for a specific substrate and if an object is outside of suchparameters—the software determines the object as “foreign.”

The identification of the insect will immediately trigger a response inan action head which is attached to the lens housing and is capable ofmoving to the insect position, and removing or destroying the insectautomatically. In another embodiment of the invention, rather thanremoving an observed insect—an action head grips the piece of lettuceand discards it.

FIG. 1 shows a scanning device having a casing 12, which houses an imagerecognition system. A downward facing microscope, lens 14 or any suchimage capturing device and magnification device is located at a bottomportion of the scanning device. As shown, the lens 14 is directed at asubstrate 16, such as, for example, a piece of lettuce. The lensmagnifies a segment of a substrate to be checked and it feeds capturedimages to an image recognition system for image analysis. Images may bestored on a digital storage medium, among other storage systems ormedia.

It will be understood by those of ordinary skill in that the device 10may be provided with a plurality of differently powered lenses which maybe automatically adjusted when greater focusing ability is needed. Itwill also be understood that the image recognition system need not behoused within the casing 12 of the device—but rather the imagerecognition software may be provided at a location that is distant fromthe image-capturing device. In such embodiment, an image-capturingdevice (e.g. a microscope lens coupled to an image capturing system) isutilized to capture images. The images are then sent by a wired orwireless connection to an image classifier.

FIG. 1 shows an action head 18, which is provided at the distal end of amovable arm 20. The action head may be equipped with one or moreinstruments, such as a gripping device and/or a suctioning device. Inanother embodiment of the invention, the action head is provided with aheating element or such similar heat source—which can destroy a bug or asegment of lettuce when it is brought into direct contact therewith.

In one embodiment of the invention, the device 10 housing the lens 14and action head 18 is a handheld unit, which may be manually orautomatically moved across a stationary substrate such as a leaf oflettuce. In another embodiment the device 10 is mounted on a stationarysupport and a conveyor belt positioned below the device delivers itemsto be scanned below the microscope lens of the device. Still in otherembodiments of the invention, the device is mounted to a linear motiontrack and it incrementally moves (for instance by incremental movementsof a rack and pinion wheel controlled by a computer) across a substrateto be searched. In one embodiment, the device 10 may be used forpurposes of “surveillance.” In this embodiment, the device is mounted ina fixed position. When an offending object (such as an insect) entersthe field of vision of the lens and is recognized as such by the imagerecognition system—a command is sent to activate the action head 18 toeliminate and/or neutralize the offending object. It will be understoodthat as an alternative to eliminating and/or neutralizing an offendingobject, the device could mark the offending object for subsequentremoval or remedial action.

It will be further understood that the moveable arm described herein maybe its own detached unit, but which operates under the control of thesoftware.

The action head 18, in an embodiment of the invention, is attached tothe distal end of a movable arm. It will be understood that a movablearm may be any of various structures such as, for example, one or morelinear guide tracks, rack and pinion systems or such similar relativemotion mechanism for supporting and moving an action head is within theteaching of the invention. The arm is movable in any of variousdirections by way of ball joints, linear motion tracks or other suchsimilar movement systems. When a bug or other offending object isdetected by the image recognition system, the software is programmed tosend a signal to the moveable arm. The moveable arm is then controlledby a software application and directed to the located bug. The actionhead is deployed to either destroy the bug as described above or tosuction it off of the substrate. In one embodiment of the invention,rather than directing the action head to a specific location—themoveable arm is directed to push the piece of lettuce (or othersubstrate) away, thereby discarding the same or removing it from abatch.

The computer used to control operations, execute routines and store datamay comprise at least one or more processors and memory storage devices.The computer also may receive a number of inputs and outputs forcommunicating information externally.

It will be understood that the computer which operated the device mayoperate under the control of an operating system and softwareapplications, components and programs that execute the routines andsystems described herein. In general, the routines executed to implementthe embodiments of the invention, whether implemented as part of anoperating system or a specific application, component, program, object,module or sequence of instructions will be referred to herein as “thesystem”, or “software”. The software controls the image acquisition,image storage, image analysis and movements of the arm, action headand/or the movement of the device along a track.

It will be understood by those of ordinary skill in the art that theinventive device is an apparatus having image capturing capabilities andimage recognition capabilities coupled with software that is programmedto determine whether or not an object in an image field is an offendingobject. An “offending object” herein is any physical, identifiablestructure or shape that is targeted for action. Examples of offendingobjects may include, but are not limited to, insects, dirt, mold growth,product imperfections etc. The device is programmed to take an actiononce an offending object is detected. “Action” refers to any remedialsteps taken by the device to eliminate or otherwise address theoffending object. For example, in one embodiment of the invention, theaction head 18 of device 10 advances to a location of an offendingobject and it records the spatial coordinates of the same. Thecoordinates are stored for later treatment and or elimination.

FIG. 2 shows a number of processing steps performed by the software inan exemplary embodiment of the invention.

The device is initiated and begins capturing images 22. The images aresent to an image recognition system which classifies various images 24.The classifier may be an algorithmic classifier or a neural networksystem. The image recognition system is trained to recognizemorphological/physical characteristics of bugs. The image recognitionmay also be trained to detect pixel concentrations which may indicatethe presence of bug or other objects of interest.

If an image is determined to be a bug 26, then the software performsfurther processing steps.

In one embodiment, the software sends a signal to the moveable arm 28,which directs the action head to the location of the bug to remove thesame according to the teachings described above.

In another embodiment of the invention, the software sends a signal tothe moveable arm to push aside the item 30 upon which the bug wasdetected.

Still in another embodiment of the invention, upon detecting a bug, thesoftware sends or sounds an alert to a human operator 32. The humanoperator may intervene to remove the bug or the item.

In another preferred embodiment, the device is designed to deal withonly one kind of insect on one kind of food or other material. A singlekind of action suitable for the situation will be built into thisdevice.

Extension is obvious to a more sophisticated device with software taughtto deal with many different kinds of foods and materials, to recognize arange of different insects, and different means of removing the insect,including vacuum, a glue head, an electrical charge, freezing, heat, oreven a drop of powerful insecticide. Powerful pesticides sprayed ordeposited over a large area is harmful, but a targeted drop on theinsect itself will dispatch the insect and not affect the surroundingatmosphere.

In another embodiment, there will be an array of lenses and responseheads so that a sheet being inspected for bed bugs can be continuouslypassed under the array.

In another embodiment of the invention the inventive system may be usedto remove offending objects, such as bugs, from a fluid. In oneembodiment, an image capturing device is fixed above a channel offlowing liquid. The device may comprise an array of image capturingdevices or lenses suspended above a channel or similar fluid stream. Itwill be understood that in an embodiment of the invention, the actionhead may be a vacuum head or suction head such that when the imagerecognition system detects a presence of an offending object, thesoftware sends a command to the action head to vacuum an area of fluidin the vicinity of the offending object. The vacuum head or suction headwill draw in the offending object, and possibly, some of the surroundingfluid and discard the same.

The invention may be used in any of various environments in which subtlechanges need to be detected and then acted on. For example, thebeginning of a disease affecting trees or other plants and itssubsequent spread is often the result of an insect, beetle or bugpenetrating the bark or other surface and destroying the structure fromwithin. Detecting a presence of a specific kind of invader is virtuallyimpossible if it requires a human observer's continued closeobservation. The instant invention can be deployed in a manner whichdetects and acts whenever an invader is detected on the surface. Forexample, the software may be trained to detect specific bugs or locusts.Once detected, the software sends a command to spray an offendingsubstance or a pesticide.

In another embodiment of the invention, the software is programmed todetect swarms of bugs or flying objects—irrespective of the type of bugsor objects. In one example, the software is trained to detect aplurality of distinct moving objects within an area of interest. Once athreshold number of moving objects (e.g. >10) is detected, the softwarewill confirm a presence of a swarm and it will automatically sendinstructions to the action head to address the swarm. In one embodiment,the action head will spray a mist of water vapor or insecticide, smokelaced with insecticide, repellant or similar offending substances.Alternatively, the device will sound an alarm to disperse the swarm.

The invention can be modified to recognize the sign of incipient diseaseon the skin of a human being at a size almost invisible to the humaneye. It is evident that the principles of the invention can be readilyapplied to other areas where detection, recognition, and action of aflaw, intrusion, or incipient flaw at a stage where it is barelyvisible.

Depending on the specific use, it is obvious that the invention can beassociated with a variety of platforms, both mobile and stationary. Forexample, the image capturing lens and action head may be mounted to amovement mechanism such as a linear guide track, a pulley system, a rackand pinion or any such similar movement mechanisms. Alternatively, thedevice may be attached to or embedded within a drone, hovercraft,aircraft or similar dirigible. (Mechanical devices/mechansims for movingthe device referred to as “movable platforms” herein.) In the embodimentwhereby the device is mounted to a movement mechanism, the software maybe programmed to move the device in any of various pre-determined orrandom movements. In an embodiment of the invention, once an offendingobject is detected, the software sends a command to interrupt movementof the device and deploy the action head to execute remedial actions.

In one preferred embodiment, the invention is attached to a drone whichis programmed to move up and down, and all around a tree periodically,subsequently to move on to other trees. This will enable the instantinvention to protect forests and plantations against invasive species.Drones can be programmed to travel in certain sectors of a forest,around a periphery or in any pattern as designated by an operator tocapture images in the area below it and instantly react.

A drone can also be programmed to remain stationary, detecting andprotecting against the arrival of an expected invasive species for whichit has been trained. Similarly, attaching the device to a robot enablesthe protection against land based invasive species.

In each case, the proliferation of robots and drones, as well as otherplatforms, extends the use and effectiveness of the device. Included insuch other platforms are hovercrafts, extendable legs and floatabledevices among others obvious to those skilled in the art.

FIG. 3 shows a drone 34 used to capture images and provide an activeresponse according to an embodiment of the invention. As shown, drone 34has a lens 36 or similar image capturing mechanism. In a preferredembodiment, the lens is housed in a movable and/or rotatable housing.The lens 36 captures images, which are then analyzed by the imageanalysis software. The image analysis software may be located in acomputer residing in the drone 34 or images may be sent via wire orwireless communication to a computer at another location.

An action arm 38 is shown extending from the body of the drone 34.Action arm 38 has one or more rotatable joints 40 a, 40 b, ball jointsor similar pivoting members allowing for various movement of the actionarm 38. For example, in the embodiment shown, action arm 38 is shownpointing downward in an orientation substantially orthogonal to the bodyof the drone, but it could be rotated around joint 40 a to a 90° angle.

An action head 42 is shown positioned at the terminal end of action arm38. The software is configured to direct action head 42 in the directionof a detected offending object and to automatically initiate remedialactions. For example, action head 42 is activated to expel any ofvarious substances described above in response to a command from thesoftware. A tank or similar reservoir within the drone stores substancesto be dispersed or dispensed from the drone.

Control of any of the above mobile or active platforms envisioned abovecan be implemented in a variety of ways, including voice recognition.

It is obvious that the insect image recognition and instant activeresponse device, as well as any attendant platform or support can besupplied with energy in a variety of ways, including batteries, solar,electromagnetic and hard wires, among others.

The instant invention is not limited to any specific materials ofconstruction or size, and is readily modified by change in programmedrecognition patters to react to different insects, and in one preferredembodiment, to detect extremely small predictable defects orimperfections, among others, in the manufacturing process or inmanufactured products.

In this latter use, it is obvious that more than one device can beconnected to work in tandem, or in any manner called for by thesituation.

In one preferred embodiment, the invention will be furnished withassisted illumination to extend its use at night, through the use oflight and infrared, among others. To extend its use further, x-ray andother surface penetrating radiation can be attached to the platform orto the device itself.

It is obvious to someone skilled in the art that the image recognitionand instant response feature can be incorporated into the platform.

In one preferred embodiment, a robot platform may be provided with animage magnification device to magnify an image of an area, segmentand/or substrate to detect objects it is trained to detect.

It will be understood that in addition to observing and capturingimages, the robot may be programmed to provide an active response toremove, mitigate and/or react to various conditions. Any of a variety ofactions steps may be deployed by the robot such as, but not limited to,sending an alert or an update, and/or expelling a spray or substancesuch as pesticide, vapor or smoke.

In a further implementation of the invention, the device is programmedto provide an instant response which results in recognition and captureinstead of recognition and reject. That is, objects, such as insects,which are of interest instead of being offending, can be trapped andcaptured with the same device, using only an obvious modification. Thatis, rather than dispersing or deflecting an object of interest, thesoftware sends a signal to the mechanical arm to capture and maintainthe object of interest.

The inventive device may be deployed for use in quality controlactivities. In this embodiment, the device may be trained to recognizequalitatively acceptable objects and those that do not meet acceptablecriteria (or “defective objects”). The software is programmed to employan action head to capture defective objects. Acceptable criteria may beany of size criteria, shape criteria or such similar metrics calculatedby the software or algorithmic classifier. In other embodiments,acceptable criteria may be based on color criteria, pixel counts, pixelsaturation or any such similar image criteria the software and imagerecognition/analysis software is programmed to analyze.

It is obvious to anyone skilled in the art that the invention can bemodified to be used as a stand-alone, hand held device, or fixed inplace with items to be inspected passing through. It is also evidentthat lens/action heads can be in a circular or other convenient pattern,and on both sides of the material, as on both sides of a sheet.

It should be understood that the preferred embodiment was described toprovide the best illustration of the principles of the invention and itspractical application to thereby enable one of ordinary skill in the artto utilize the invention in various embodiments and with variousmodifications as are suited to the particular use contemplated.

What is claimed is:
 1. A device for detecting offending objects,comprising: an image capturing device; an image recognition systemconnected to said image capturing device, said image recognition systembeing trained to recognize characteristics of said offending object; amoveable arm, said moveable arm deployable to contact a substrate thatis being searched for offending objects; an action head provided on saidmoveable arm; software configured to detect a presence of an offendingobject and to control said moveable arm to move to said action head tosaid offending object; said device for detecting offending objectsdisposed on a movable platform.
 2. The device of claim 1, whereby saidmovable platform is a linear guide track.
 3. The device of claim 1,where said movable platform is a drone.
 4. The device of claim 1,whereby said movable platform is a dirigible.
 5. The device of claim 1,further comprising software configured to control movement of saidmovable platform.
 6. The device of claim 5, whereby said software isconfigured to interrupt said movement of said movable platform when anobject of interest is detected.
 7. The device of claim 1, whereby saidaction head is configured to grip an offending object.
 8. The device ofclaim 1, whereby said software is configured to command said action headto disperse a substance when an offending object is detected.